Skip to content

[sanitizer_common] [Darwin] Adopt _dyld_get_dyld_header - #182943

Merged
ndrewh merged 1 commit into
llvm:mainfrom
ndrewh:GetDyldImageHeaderViaSharedCache
Feb 24, 2026
Merged

ndrewh merged 1 commit into
llvm:mainfrom
ndrewh:GetDyldImageHeaderViaSharedCache

Conversation

@ndrewh

@ndrewh ndrewh commented Feb 23, 2026 •

Copy link
Copy Markdown
Contributor

There is an issue on recent macOS versions with GetDyldImageHeaderViaSharedCache, which is fixed by adopting _dyld_get_dyld_header. We weakly declare this to ensure runtimes compile with older SDK (currently on CI bots).

rdar://167854578

@llvmbot

llvmbot commented Feb 23, 2026

Copy link
Copy Markdown
Member

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Andrew Haberlandt (ndrewh)

Changes

There is an issue on recent macOS versions with GetDyldImageHeaderViaSharedCache, which is fixed by adopting _dyld_get_dyld_header. We weakly declare this to ensure runtimes work on older OS, and also newer OS compiled with older SDK (i.e. the current bots).

rdar://167854578


Full diff: https://github.com/llvm/llvm-project/pull/182943.diff

3 Files Affected:

  • (modified) compiler-rt/lib/sanitizer_common/sanitizer_procmaps_mac.cpp (+6-2)
  • (modified) compiler-rt/lib/sanitizer_common/weak_symbols.txt (+1)
  • (modified) compiler-rt/lib/tsan/go/buildgo.sh (+1-1)
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_mac.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_mac.cpp
index 979729f15aa16..0e80f2e3916b2 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_mac.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_procmaps_mac.cpp
@@ -176,7 +176,7 @@ void MemoryMappingLayout::Reset() {
 // The dyld load address should be unchanged throughout process execution,
 // and it is expensive to compute once many libraries have been loaded,
 // so cache it here and do not reset.
-static mach_header *dyld_hdr = 0;
+static const mach_header *dyld_hdr = 0;
 static const char kDyldPath[] = "/usr/lib/dyld";
 static const int kDyldImageIdx = -1;
 
@@ -244,14 +244,18 @@ extern intptr_t _dyld_get_image_slide(const struct mach_header* mh);
 extern int dyld_shared_cache_iterate_text(
     const uuid_t cacheUuid,
     void (^callback)(const dyld_shared_cache_dylib_text_info *info));
+SANITIZER_WEAK_IMPORT const struct mach_header *_dyld_get_dyld_header(void);
 }  // extern "C"
 
-static mach_header *GetDyldImageHeaderViaSharedCache() {
+static const mach_header *GetDyldImageHeaderViaSharedCache() {
   uuid_t uuid;
   bool hasCache = _dyld_get_shared_cache_uuid(uuid);
   if (!hasCache)
     return nullptr;
 
+  if (&_dyld_get_dyld_header != nullptr)
+    return _dyld_get_dyld_header();
+
   size_t cacheLength;
   __block uptr cacheStart = (uptr)_dyld_get_shared_cache_range(&cacheLength);
   CHECK(cacheStart && cacheLength);
diff --git a/compiler-rt/lib/sanitizer_common/weak_symbols.txt b/compiler-rt/lib/sanitizer_common/weak_symbols.txt
index 77e7b5d9f702e..600ab8a7c649c 100644
--- a/compiler-rt/lib/sanitizer_common/weak_symbols.txt
+++ b/compiler-rt/lib/sanitizer_common/weak_symbols.txt
@@ -10,3 +10,4 @@ ___sanitizer_symbolize_demangle
 ___sanitizer_symbolize_flush
 ___sanitizer_symbolize_set_demangle
 ___sanitizer_symbolize_set_inline_frames
+__dyld_get_dyld_header
diff --git a/compiler-rt/lib/tsan/go/buildgo.sh b/compiler-rt/lib/tsan/go/buildgo.sh
index d9e56402ad48f..1340071819fcb 100755
--- a/compiler-rt/lib/tsan/go/buildgo.sh
+++ b/compiler-rt/lib/tsan/go/buildgo.sh
@@ -165,7 +165,7 @@ elif [ "$GOOS" = "netbsd" ]; then
 	"
 elif [ "$GOOS" = "darwin" ]; then
 	OSCFLAGS="-fPIC -Wno-unused-const-variable -Wno-unknown-warning-option -mmacosx-version-min=10.7"
-	OSLDFLAGS="-lpthread -fPIC -fpie -mmacosx-version-min=10.7"
+	OSLDFLAGS="-lpthread -fPIC -fpie -mmacosx-version-min=10.7 -Wl,-U,__dyld_get_dyld_header"
 	SRCS="
 		$SRCS
 		../rtl/tsan_platform_mac.cpp

@github-actions

github-actions Bot commented Feb 23, 2026 •

Copy link
Copy Markdown

✅ With the latest revision this PR passed the C/C++ code formatter.

@ndrewh
ndrewh force-pushed the GetDyldImageHeaderViaSharedCache branch from ea41f5f to bdf57bf Compare February 23, 2026 21:44
@ndrewh

ndrewh commented Feb 23, 2026

Copy link
Copy Markdown
Contributor Author

Fixed clang-format

@ndrewh
ndrewh merged commit 2e7d07a into llvm:main Feb 24, 2026
10 checks passed
gopherbot pushed a commit to golang/go that referenced this pull request Mar 19, 2026
With llvm/llvm-project#182943, the race
detector syso has a weak import of __dyld_get_dyld_header, which
is only defined on newer macOS (26.4+). For external linking with
a pre-Xcode 26.4 C toolchain, we need to tell the C linker to
permit that symbol not being defined. Pass a flag to do so.

Change-Id: I95a3cd2c7fd3ad50bc47985b3ecca0d4e8352162
Reviewed-on: https://go-review.googlesource.com/c/go/+/755261
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Lasse Folger <lassefolger@google.com>
@ndrewh

ndrewh commented Mar 27, 2026

Copy link
Copy Markdown
Contributor Author

/cherry-pick 2e7d07a

@ndrewh

ndrewh commented Mar 27, 2026

Copy link
Copy Markdown
Contributor Author

I think we should cherry-pick this to make sure that clang 22 supports ASAN on macOS 26.4.

@llvmbot

llvmbot commented Mar 27, 2026

Copy link
Copy Markdown
Member

/cherry-pick 2e7d07a

Error: Command failed due to missing milestone.

@ndrewh ndrewh added this to the LLVM 22.x Release milestone Mar 27, 2026
@github-project-automation github-project-automation Bot moved this to Needs Triage in LLVM Release Status Mar 27, 2026
@github-project-automation github-project-automation Bot moved this from Needs Triage to Done in LLVM Release Status Mar 27, 2026
@ndrewh

ndrewh commented Mar 27, 2026

Copy link
Copy Markdown
Contributor Author

/cherry-pick 2e7d07a

@llvmbot

llvmbot commented Mar 27, 2026

Copy link
Copy Markdown
Member

/pull-request #188913

dylan-conway added a commit to oven-sh/bun that referenced this pull request Mar 30, 2026
…t mode

DirectBuild: deps simple enough to list files compile as first-class
ninja edges instead of a cmake sub-process. tinycc converted — drops
the overlay CMakeLists.txt and the recurring ASAN workarounds on the
c2str host tool, which now gets -fno-sanitize=all unconditionally.

ASAN dyld shim: macOS 26.4's Dyld.framework reimplemented
dyld_shared_cache_iterate_text in Swift with a _Block_copy that
deadlocks ASAN init (llvm/llvm-project#182943). Shim interposes a
non-allocating version using _dyld_get_dyld_header. Linked via
LC_LOAD_DYLIB + @rpath so it auto-loads with bun-debug — no env var.
Darwin+ASAN only.

workarounds.ts: self-obsoleting registry. Every temporary fix registers
an expectedToBeFixed predicate; configure fails with cleanup
instructions once the upstream fix ships.

build.ts: quiet build output when exec args present (original bd
behavior), `--` separator for flag disambiguation, `--target` accepts
space-separated value, colored target name in done message, signal
exit re-raised. Arg routing documented in file header.

tty.ts: centralized ANSI/TTY abstraction. nameColor() hashes names to
stable colors; stream.ts and the done message share it.

config.ts/tools.ts: clangVersion captured from the existing
toolchain-resolve --version spawn, exposed on Config. findTool returns
{path, version}.

Also:
- configure() no longer prints; build.ts decides based on quiet mode
- provides.sources deps get phonies pointing at compiled .o files
- scripts/bd, scripts/bd.ps1 removed (bd is a package.json script)
- scripts/build/CLAUDE.md: architecture doc with goals, ninja primer,
  common tasks, arg routing
c-rhodes pushed a commit to llvmbot/llvm-project that referenced this pull request Mar 31, 2026
ndrewh referenced this pull request in modular/modular Apr 12, 2026
past the ASAN runtime fix (llvm/llvm-project#191039).

MODULAR_ORIG_COMMIT_REV_ID: 1773da5eabdc1456e1fac443ccf0aac439c6a895
dylan-conway pushed a commit to oven-sh/bun that referenced this pull request Jul 17, 2026
…macOS (#34508)

## Problem

`bun bd` aborts at startup on macOS releases older than 26.4 (seen on
15.6.1, Homebrew llvm@21):

```
AddressSanitizer: CHECK failed: sanitizer_procmaps_mac.cpp:214 "((res)) == ((0))" (0xffffffffffffffff, 0x0)
    <empty stack>
Abort trap: 6
```

The post-link `bun-debug --revision` smoke test dies with this, so the
debug build never completes. Workaround has been `bun run build
--asan=off`.

## Cause

`scripts/build/shims/asan-dyld-shim.c` interposes
`dyld_shared_cache_iterate_text` to keep ASAN init from deadlocking on
macOS 26.4 (llvm/llvm-project#182943). It looks up the private
`_dyld_get_dyld_header` via `dlsym` to synthesize the one cache entry
ASAN needs.

Apple's own `dyld_priv.h` marks that symbol ["Added in macOS/iOS
26.4"](https://github.com/apple-oss-distributions/dyld/blob/dyld-1376.6/include/mach-o/dyld_priv.h)
(first appears in dyld-1376.6; absent from dyld-1340 and every earlier
tag). On any macOS < 26.4, `dlsym` returns `NULL`, the shim returns
`-1`, and compiler-rt's `CHECK_EQ(res, 0)` aborts.

The shim is linked into every `cfg.darwin && cfg.asan` build with no
runtime check, so it breaks every debug build on macOS hosts that
predate 26.4.

## Fix

When the shim can't synthesize the entry (`dlsym` returned `NULL`, or no
shared cache), delegate to the real `dyld_shared_cache_iterate_text`
instead of returning `-1`. The deadlock and the getter were introduced
together in 26.4, so "getter absent" is exactly "real iterate is safe".

This is the same shape as the upstream compiler-rt fix
(llvm/llvm-project#188913): weak-import `_dyld_get_dyld_header`, use it
when present, otherwise fall through to the existing iterate path.

Calling the original by name from inside the interposer does not
recurse. dyld explicitly adds an identity-mapping tuple for the defining
image; from
[DyldRuntimeState.cpp](https://github.com/apple-oss-distributions/dyld/blob/dyld-1378/dyld/DyldRuntimeState.cpp):

> `// now add specific interpose so that the generic is not applied to
the interposing dylib, so it can call through to old impl`

(This is also how Apple's own `DYLD_INTERPOSE` wrapper example in
[dyld-interposing.h](https://github.com/apple-oss-distributions/dyld/blob/main/include/mach-o/dyld-interposing.h)
works.)

## Verification

Shim is only compiled for `darwin && asan` (no CI lane exists for that
combination); verified the change compiles clean with `-Wall -Wextra
-fblocks` against stubbed darwin headers. Behaviour on 26.4+ is
unchanged: `dlsym` finds the symbol there and the same synthesize path
runs as before.

## Why this is the right fix

The alternative is gating shim emission on host macOS version in
`shims.ts`, but the runtime check is strictly better: one binary works
on both, and it matches upstream exactly. The shim is temporary
regardless (self-obsoletes via `workarounds.ts` once LLVM 22.1.4+ is the
floor; #34299 deletes it as part of the LLVM 22 bump), so the goal here
is just to stop breaking pre-26.4 macOS until that lands.

<!-- robobun:evidence:begin -->

---

**no test proof** · iteration 0 · build/CI scripts only; test-proof not
applicable

<!-- robobun:evidence:end -->
liooil pushed a commit to liooil/poly that referenced this pull request Aug 7, 2026
…macOS (#34508)

## Problem

`bun bd` aborts at startup on macOS releases older than 26.4 (seen on
15.6.1, Homebrew llvm@21):

```
AddressSanitizer: CHECK failed: sanitizer_procmaps_mac.cpp:214 "((res)) == ((0))" (0xffffffffffffffff, 0x0)
    <empty stack>
Abort trap: 6
```

The post-link `bun-debug --revision` smoke test dies with this, so the
debug build never completes. Workaround has been `bun run build
--asan=off`.

## Cause

`scripts/build/shims/asan-dyld-shim.c` interposes
`dyld_shared_cache_iterate_text` to keep ASAN init from deadlocking on
macOS 26.4 (llvm/llvm-project#182943). It looks up the private
`_dyld_get_dyld_header` via `dlsym` to synthesize the one cache entry
ASAN needs.

Apple's own `dyld_priv.h` marks that symbol ["Added in macOS/iOS
26.4"](https://github.com/apple-oss-distributions/dyld/blob/dyld-1376.6/include/mach-o/dyld_priv.h)
(first appears in dyld-1376.6; absent from dyld-1340 and every earlier
tag). On any macOS < 26.4, `dlsym` returns `NULL`, the shim returns
`-1`, and compiler-rt's `CHECK_EQ(res, 0)` aborts.

The shim is linked into every `cfg.darwin && cfg.asan` build with no
runtime check, so it breaks every debug build on macOS hosts that
predate 26.4.

## Fix

When the shim can't synthesize the entry (`dlsym` returned `NULL`, or no
shared cache), delegate to the real `dyld_shared_cache_iterate_text`
instead of returning `-1`. The deadlock and the getter were introduced
together in 26.4, so "getter absent" is exactly "real iterate is safe".

This is the same shape as the upstream compiler-rt fix
(llvm/llvm-project#188913): weak-import `_dyld_get_dyld_header`, use it
when present, otherwise fall through to the existing iterate path.

Calling the original by name from inside the interposer does not
recurse. dyld explicitly adds an identity-mapping tuple for the defining
image; from
[DyldRuntimeState.cpp](https://github.com/apple-oss-distributions/dyld/blob/dyld-1378/dyld/DyldRuntimeState.cpp):

> `// now add specific interpose so that the generic is not applied to
the interposing dylib, so it can call through to old impl`

(This is also how Apple's own `DYLD_INTERPOSE` wrapper example in
[dyld-interposing.h](https://github.com/apple-oss-distributions/dyld/blob/main/include/mach-o/dyld-interposing.h)
works.)

## Verification

Shim is only compiled for `darwin && asan` (no CI lane exists for that
combination); verified the change compiles clean with `-Wall -Wextra
-fblocks` against stubbed darwin headers. Behaviour on 26.4+ is
unchanged: `dlsym` finds the symbol there and the same synthesize path
runs as before.

## Why this is the right fix

The alternative is gating shim emission on host macOS version in
`shims.ts`, but the runtime check is strictly better: one binary works
on both, and it matches upstream exactly. The shim is temporary
regardless (self-obsoletes via `workarounds.ts` once LLVM 22.1.4+ is the
floor; #34299 deletes it as part of the LLVM 22 bump), so the goal here
is just to stop breaking pre-26.4 macOS until that lands.

<!-- robobun:evidence:begin -->

---

**no test proof** · iteration 0 · build/CI scripts only; test-proof not
applicable

<!-- robobun:evidence:end -->
helly25 added a commit to bazel-contrib/toolchains_llvm that referenced this pull request Aug 16, 2026
Fixes #826.

## Summary

Wire rules_cc's stock `lsan` feature into the same augmentation path as
asan, ubsan, and tsan:

- add `//toolchain/config:use_lsan` and include it in
`use_common_sanitizer`
- add the Darwin `libclang_rt.lsan_osx_dynamic.dylib` filegroup and
route it through `dynamic_runtime_lib`
- add an end-to-end `//:lsan_test` and run it with the Linux sanitizer
tests
- add a focused macOS CI job using LLVM 22.1.8

The Darwin version pin is deliberate. On current macOS, LLVM 19.1.7,
20.1.8, and 21.1.8 all run `main` and then deadlock during LSan's exit
check. The leak-check thread calls `dyld_shared_cache_iterate_text`,
which lazily loads dyld introspection code and allocates while LSan
holds its allocator lock. LLVM compiler-rt #182943 switched this path to
`_dyld_get_dyld_header`; that fix was backported to LLVM 22, and LLVM
22.1.8 exits normally.

The configured action graph now contains a `SolibSymlink` whose declared
input is `libclang_rt.lsan_osx_dynamic.dylib`, so the runtime is
provisioned rather than merely found incidentally in the local execroot.

## Verification

On macOS arm64:

- LLVM 19.1.7: times out after completing `main`
- LLVM 20.1.8: times out after completing `main`
- LLVM 21.1.8: times out after completing `main`
- LLVM 22.1.8: `//:lsan_test` passes
- `//:sanitizer_combo_flags_test` passes
- `actionlint` and `git diff --check` pass

Upstream fix: llvm/llvm-project#182943
yiguolei pushed a commit to apache/doris that referenced this pull request Sep 30, 2026
…OS 26.4+ (#68595)

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache/doris-website#4180 (the macOS build guide change that
goes with this PR)

Problem Summary:

**Context.**

On macOS the only entry point for the BE unit tests is `run-be-ut.sh`,
and it defaults to `BUILD_TYPE_UT=ASAN`. Since macOS 26.4 that default
cannot start a single process: every binary linked against llvm.org's
`libclang_rt.asan_osx_dynamic` deadlocks during runtime initialisation,
before `main()`. `env.sh` selects that same toolchain on macOS
(`llvm@20`), so a developer who follows the documented setup gets a
build that hangs rather than one that fails.

How the macOS toolchain gets chosen matters for the rest of this
description:

- **Selection.** `env.sh` writes `custom_env_mac.sh`, which puts `$(brew
--prefix)/opt/<formula>/bin` on `PATH` for every formula in its
`CELLARS` list. Unless `DORIS_CLANG_HOME` is already set, `env.sh` then
takes the first `clang` on `PATH` and derives `CC`/`CXX` from it.
- **Installation.** `CELLARS` only selects; it installs nothing.
Developers install Homebrew formulae by following the macOS build guide,
and each CI workflow installs from its own `brew install` list.
- **Third-party libraries.** `thirdparty/installed` comes from one of
two places:
- the published `doris-thirdparty-prebuilt-darwin-*.tar.xz`, which
`apache/doris-thirdparty`'s `build-target.yml` builds by running
`thirdparty/build-thirdparty.sh` from apache/doris master;
- a local run of that same script. `build.sh` also starts one by itself
when `thirdparty/installed` lacks a library it checks for.

**1. The problem, and what it cost**

*The deadlock.* `sample` on the stalled process shows the whole chain:

```
__malloc_init (libsystem_malloc)                      <- early libSystem init calls malloc_default_zone()
 `- wrap_malloc_default_zone (asan runtime)
     `- AsanInitFromRtl -> AsanInitInternal -> InitializeShadowMemory
         `- MemoryRangeIsAvailable -> MemoryMappingLayout::Next -> get_dyld_hdr()
             `- dyld_shared_cache_iterate_text_swift   <- macOS 26.4 reimplemented this in Swift
                 `- _Block_copy -> malloc
                     `- __sanitizer_mz_malloc (asan's own malloc)
                         `- AsanInitFromRtl()              <- re-enters init
                             `- StaticSpinMutex::LockSlow  <- spins on the lock it already holds
```

The cause is an OS-side change, not a Doris one. It also does not mean
ASAN is broken on macOS 26: Apple's own clang sanitizer runtime runs
fine on the same host, and only llvm.org's compiler-rt is affected.
Upstream fixed it by weak-importing `_dyld_get_dyld_header` and using
it, when present, instead of walking the shared cache
(llvm/llvm-project#182943, main `2e7d07a`; backport #188913,
release/22.x `7b6514c`). **The fix shipped only in 22.1.8.** 20.1.8 is
the last 20.x release and 21.1.8 the last 21.x, so no version of
`llvm@20` can ever be made to work.

*Why it looks like a hang and not an error.*

- `be/CMakeLists.txt` reaches `storage/index/ann` unconditionally, and
that directory's `cmake-protect` target calls `add_subdirectory()` on
`contrib/openblas`.
- OpenBLAS runs an instrumented `getarch` probe from its **configure**
step (`contrib/openblas/cmake/prebuild.cmake:1513`,
`execute_process(COMMAND .../getarch 0 ...)`), and `execute_process` has
no timeout.
- So every ASAN build stops at `-- Running getarch` and never moves
again. It sits at about 90% CPU and prints no diagnostic.

`run-be-ut.sh` on macOS was therefore unusable at its default setting,
and the failure gave the developer nothing to act on.

*The second obstacle, once the toolchain is bumped.*

- clang 22 added `-Wc2y-extensions` and folds it into `-Wpedantic`.
- `__COUNTER__` only reached the C standard in C2y (WG14 N3457).
`be/src/runtime/runtime_profile.h:73-85` uses it to give two
`SCOPED_TIMER` / `SCOPED_RAW_TIMER` expansions on the same line distinct
names.
- As a result, a clang 22 build of any TU that includes that header
fails under `-Werror`. Three representative unity TUs were enough to hit
it, so the failure is not confined to one module.

*The third obstacle: clang 22 cannot build the third-party libraries.*
The first revision of this PR moved every macOS toolchain reference to
`llvm@22`, including the third-party build. CI failed both macOS
third-party jobs (7m49s, 6m23s) on the first library that exercises a
new clang 22 error:

```
unixODBC 2.3.7  src/SQLBrowseConnectW.c:424:82:
error: incompatible pointer types passing 'SQLSMALLINT *' (aka 'short *')
       to parameter of type 'int *' [-Wincompatible-pointer-types]
```

clang 16, clang 20 and Apple clang 21 report this as a warning; clang 22
makes it an error. unixODBC is only the *first* failure: the build
aborts there, so every library after it is untested against clang 22.
The macOS third-party libraries therefore have to stay on `llvm@20`
while the BE moves to `llvm@22`.

Leaving the third-party build alone does not achieve that, because it
sources `env.sh`:

- With `CELLARS` naming `llvm@22`, a machine without `llvm@22` gets a
non-existent directory on `PATH`.
- `command -v clang` then falls through to `/usr/bin/clang` (Apple
clang), which is worse than either explicit choice.
- The job that publishes the prebuilt archive in
`apache/doris-thirdparty` is exactly such a machine: it installs only
`llvm@20` and sets no `DORIS_CLANG_HOME`.

*Two latent defects clang 22 then surfaced.* The bump is not a pure
version change. clang 22's stricter diagnostics stop the build on two
pre-existing bugs, and both are worth fixing on their own merits.

| # | Site | Diagnostic | What is actually wrong |
|---|---|---|---|
| 1 | `be/src/load/group_commit/wal/wal_dirs_info.cpp:98` |
`-Wunused-result` | `LOG(INFO) << "… err: {}", e.what();` — the `,` is
the comma operator, not an argument separator, so the statement is
`(LOG(INFO) << "…{}") , (e.what())`. The `{}` is never substituted, and
`e.what()` is evaluated and then discarded: **the error message has
never been logged**, only the literal `{}`. A repo-wide scan for the
same shape finds exactly this one site (`cloud/src/common/logging.h`
matches only inside macro definitions and is not a bug). |
| 2 | `common/cpp/sync_point.cpp:208,210` | `-Wthread-safety-analysis` |
The function holds `std::unique_lock lock(mutex_)`, then releases and
re-takes the mutex with a raw `mutex_.unlock()` / `mutex_.lock()` pair
around the callback, which bypasses the lock's ownership tracking. If
the callback throws, the re-lock is skipped while `~unique_lock` still
believes it owns the mutex, so the destructor unlocks a mutex that is
not held (UB) and `num_callbacks_running_` is never decremented.
`lock.unlock()` / `lock.lock()` still runs the callback unlocked, as
intended, and keeps the ownership state correct on every path. |

**2. What this PR does, and why it helps**

- **`env.sh`**: the macOS `CELLARS` list moves from `llvm@20` to
`llvm@22`, with a comment recording that 22.1.8 is the minimum and why.
This is the single line that decides which clang a macOS developer
builds the BE with.
- **`thirdparty/build-thirdparty.sh`**: on Darwin, right after sourcing
`env.sh` and next to the existing environment sanitization, the script
unconditionally points `DORIS_CLANG_HOME`, `CC` and `CXX` at `$(brew
--prefix llvm@20)` and puts its `bin/` first on `PATH`. If `llvm@20` is
not installed, it stops with a `brew install llvm@20` hint rather than
falling back to another compiler.
- The override has to come after `env.sh` and has to be unconditional.
`env.sh` sources `custom_env.sh`, which may export `DORIS_CLANG_HOME`
for the BE, and `build.sh` has already exported the BE's `llvm@22`
values before it starts this script.
- Every macOS third-party build runs this script: `build.sh`'s automatic
rebuild, a manual run, the pull request check in `build-thirdparty.yml`,
and the `apache/doris-thirdparty` job that publishes
`doris-thirdparty-prebuilt-darwin-*.tar.xz`. This one place therefore
keeps all of them, and the published archive, on `llvm@20`.
- `.github/workflows/build-thirdparty.yml` is unchanged: its macOS jobs
already install `llvm@20`.
- **`be/CMakeLists.txt`**: add `-Wno-c2y-extensions` for clang 19 and
newer, in its own `add_compile_options` call **after** `-Wpedantic`.
- The position matters: a later `-Wpedantic` turns the group back on.
That is also why passing the flag through `EXTRA_CXX_FLAGS` does not
work: that variable lands near the front of the command line.
- The version gate matters because the `c2y-extensions` group only
exists from clang 19 on. clang rejects an unknown `-Wno-` option like
any other unknown warning option (`-Wunknown-warning-option`, fatal
under `-Werror`), and this file accepts clang 16 and newer.
  - For clang 19 and newer the command line is exactly what it was.
- **`.github/workflows/be-ut-mac.yml`**: install `llvm@22` for the BE
and keep `llvm@20` next to it.
- The job builds the BE on top of the downloaded darwin-arm64 prebuilt.
When that archive lacks a library `build.sh` checks for, `build.sh`
rebuilds the third-party libraries from source, and that rebuild now
needs `llvm@20`. The archive falls behind like this whenever master
changes the set of checked libraries before the next archive is
published.
- Both formulae are keg-only, so installing both does not change which
one the BE uses.
- **`be/src/load/group_commit/wal/wal_dirs_info.cpp` and
`common/cpp/sync_point.cpp`**: the two fixes from the table above. Each
is one line, the smallest change that removes the defect rather than
suppressing the warning. No `-Wno-unused-result` /
`-Wno-thread-safety-analysis` is added, because those diagnostics point
at real bugs and should keep firing.
- **apache/doris-website#4180**: the macOS build guide installs
`llvm@22` and `llvm@20` instead of only `llvm@20`. Without it, a
developer who follows the guide after this PR has no `llvm@22`, and the
BE silently builds with Apple clang.

What this PR deliberately does not do is add `llvm@20` to `env.sh`'s
`CELLARS`:

- `CELLARS` installs nothing.
- `build-thirdparty.sh` does not use `PATH` to find `llvm@20`.
- The `CELLARS` loop prepends each entry to `PATH`. An `llvm@20` entry
after `llvm@22` would put the BE back on `llvm@20` and bring the
deadlock back.

**3. The classes, and how they call each other**

```
env.sh  CELLARS := llvm@22                                (selects a compiler; installs nothing)
  |- generates custom_env_mac.sh: $(brew --prefix)/opt/<cellar>/bin prepended to PATH
  |- DORIS_CLANG_HOME := dirname($(command -v clang))/..   -> CC / CXX / ASAN_SYMBOLIZER_PATH
  '- is sourced by:
       |- build.sh, run-be-ut.sh                           -> BE: llvm@22
       '- thirdparty/build-thirdparty.sh:55
            '- :100-110  on Darwin: DORIS_CLANG_HOME / CC / CXX := $(brew --prefix llvm@20)
                         llvm@20 missing -> exit 1, "brew install llvm@20"
                                                           -> third-party libraries: llvm@20
                 run by: build.sh (library missing from thirdparty/installed), manual runs,
                         build-thirdparty.yml (PR check), doris-thirdparty build-target.yml (publisher)

brew install lists, i.e. who installs which LLVM:
  macOS build guide (apache/doris-website#4180)   llvm@22 llvm@20
  .github/workflows/be-ut-mac.yml                 llvm@22 llvm@20
  .github/workflows/build-thirdparty.yml          llvm@20
  apache/doris-thirdparty build-target.yml        llvm@20

be/CMakeLists.txt  if (COMPILER_CLANG)
  |- add_compile_options(-Wpedantic ...)               <- enables the c2y group
  '- if (CMAKE_CXX_COMPILER_VERSION >= 19)
       add_compile_options(-Wno-c2y-extensions)        <- must stay after the line above
         |
be/src/runtime/runtime_profile.h:73-85
  '- MACRO_CONCAT(SCOPED_TIMER, __COUNTER__)           <- the only __COUNTER__ use in be/src, be/test
         |
be/src/storage/index/ann/cmake-protect/CMakeLists.txt:48
  '- add_subdirectory(contrib/openblas)
       '- cmake/prebuild.cmake:1513  execute_process(getarch)   <- where the hang surfaced
```

### Release note

None

### Check List (For Author)

- Test: Manual test on macOS 26.5.1 (arm64, dyld-1378), plus CI.
- `clang -fsanitize=address` hello world exits 0 with llvm@22 and still
deadlocks with llvm@20 on the same host.
- `sh run-be-ut.sh` with no environment overrides now selects
`Clang-22.1.8` and clears the `-- Running getarch` point that previously
hung forever: zero FAILED targets, and `doris_be_test` links.
- `sh run-be-ut.sh --run --filter=FormatRoundTest.*` starts the ASAN
binary and passes 8 tests.
- `-Wno-c2y-extensions` gate, tested on a `__COUNTER__` TU under
`-Wpedantic -Werror`: clang 16.0.6 rejects the flag (`unknown warning
option`), which is why it is gated; clang 20.1.8 accepts it; clang
22.1.8 fails without it and passes with it.
- `build-thirdparty.sh` compiler override, checked under `bash -x` with
both `custom_env.sh` and `build.sh` exporting the `llvm@22` values: the
effective `CC`/`CXX` and the first clang on `PATH` are llvm@20. With
`llvm@20` missing, the script exits 1 with the brew hint before building
anything.
- `be-ut-mac.yml`: the "Build BE" step script passes `bash -n` on bash
3.2 and 5.3, and its `cellars` array holds both `llvm@22` and `llvm@20`.
The workflow runs only on pushes to master and on schedule, so this PR
cannot run it.
    - `build-support/check-build-hygiene.sh` passes.
- CI: `Build Third Party Libraries (macOS)`, `(macOS-arm64)` and
`(Linux)` pass at `a955357a30f` (run 36542107899), which already
contains the `build-thirdparty.sh` override. The first revision, which
moved these builds to llvm@22, failed both macOS jobs on unixODBC.
- Regression test / unit test: N/A (toolchain, flag and workflow
change).
- Behavior changed: Yes, for macOS builds only.
    - The BE builds with llvm@22 instead of llvm@20.
- The third-party libraries, including the published prebuilt, stay on
llvm@20. Building them from source now requires llvm@20 installed next
to llvm@22; without it, the build stops with a `brew install llvm@20`
hint.
- Existing macOS developers need to run `brew install llvm@22`. Without
it, `env.sh` falls back to Apple clang in `/usr/bin` for newly
configured build directories.
- Does this need documentation: Yes, apache/doris-website#4180.

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

3 participants